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Executive  Summary 


This  paper  describes  two  analytic  tools  for  enhancing  the  foundational  analysis  and 
the  mission  effectiveness  outcome  on  major  defense  acquisition  programs.  The  first  tool 
is  Mission  Stream  Analysis,  which  aids  in  decomposing  the  mission  and  system 
capabilities/requirements  into  system  technical  performance,  operator  workload 
requirements,  and  metrics  that  contribute  to  demonstrating  mission  effectiveness.  The 
second  tool  is  the  A  (Delta)  Analytic  Model,  which  provides  an  approach  for  identifying 
disparate  interpretations  of  the  systems  requirements  and  metrics  in  the  analytic 
foundation  so  that  the  differences  can  be  eliminated. 

Mission  Stream  Analysis  is  conducted  on  a  system  for  a  specific  mission  scenario.  It 
decomposes  a  mission  scenario  into  a  series  of  offensive  and  defensive  kill-chains, 
survivability  spectra,  and  other  mission  tasks.  With  each  task  is  associated  the  capabilities 
and  mission  systems  needed  to  perfonn  the  task.  The  missions,  so  modeled,  provide  a 
framework  for  the  communities  that  participate  in  establishing  the  requirements,  metrics, 
and  tests  that  will  demonstrate  the  system  can  effectively  complete  the  missions  for 
which  it  is  intended.  We  believe  the  effectiveness  measures  that  are  output  from  the 
mission  stream  analyses  are  as  important  to  a  program’s  success  as  the  Key  Perfonnance 
Parameters. 

The  A  Analytic  Model  seeks  to  eliminate  disparate  interpretations  of  the  systems 
requirements  and  metrics,  by  identifying  differences,  in  the  Analytic  Foundation  of  the 
program.  The  A  Analytic  Model  is  the  “domain-centric”  environment  in  which 
communities  reach  down  to  the  Analytic  Foundation  for  key  elements  of  information, 
regardless  of  the  time  phasing  of  the  program.  For  the  A  Analytic  Model  to  work,  the 
Analytic  Foundation  must  be  transparent  and  readily  available  to  all  communities,  and  all 
communities  must  recognize  the  Analytic  Foundation  as  the  authoritative  source  of 
information.  We  envision  a  model  that  is  not  unlike  Wikipedia  with  permissions. 
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1.  Introduction 


Weapon  system  program  development  problems,  as  evidenced  by  significant  cost 
increases  and  schedule  delays,  highlight  the  need  to  improve  both  discipline  and 
oversight  in  the  Department  of  Defense  (DoD)  acquisition  process.  Senator  John  McCain 
best  articulated  these  problems  in  his  comments  on  February  4,  2013: 1 

There  are  far  too  many  examples  where  the  Department  begins  a  major 
program  without  knowing  what  it  really  wants  or  how  these  requirements 
should  translate  into  technical  specifications  that  are  designed  to  generate 
the  combat  capability  it  really  needs.  Also,  all  too  many  times,  there  is  no 
traceability  between  these  specifications  through  a  test  regime  that  is 
sufficient  to  ensure  that  the  system  the  Department  is  procuring  is 
operationally  effective,  suitable  and  survivable  before  entering  operational 
testing  or  early  production. 

To  address  these  issues,  this  paper  proposes  a  methodology  based  on  well-defined 
“system”  and  “mission”  requirements  that  (1)  enables  the  development  and  execution  of 
an  efficient  and  effective  design  and  validation  process,  and  (2)  provides  confidence  that 
the  performance  of  a  system  is  understood  and  developed  in  the  mission  context.  The 
proposed  methodology  includes: 

•  A  “A  Analytic  Model”  that  provides  for  better  cross-domain  (requirements, 
systems  engineering,  and  test  and  evaluation)  collaboration  and  analysis.  The 
model  helps  reduce  definitional  conflicts  and  aligns  development  expectations 
across  domains;  this  model  is  a  companion  to  “Mission  Stream  Analysis.” 

•  “Mission  Stream  Analysis”  is  based  on  the  concept  that  weapon  systems  are 
designed  around  the  synergy  between  man  (operator’s  capability)  and  machine 
(system’s  technical  performance)  to  effectively  accomplish  a  mission.  In 
essence,  mission  stream  analysis  is  a  tool  for  defining,  developing,  and 
evaluating  a  weapon  system’s  capability  to  obtain,  integrate,  analyze,  share,  and 
act  on  information  within  the  operator’s  decision  cycle  in  order  to  effectively 


“Floor  Statement  by  Senator  McCain  on  the  Defense  Department’s  ‘Culture  of  Inefficiency’  and 
Senator  Hagel’s  Nomination,”  John  McCain,  U.S.  Senator,  Arizona, 

http://www.mccain.senate.gov/public/index.cfm/floor-statements?ID=a7256332-b509-07dd-5300- 

18230cfbfe04. 

“The  PM  shall  apply  human  systems  integration  to  optimize  total  system  performance  (hardware, 
software,  and  human),  operational  effectiveness,  and  suitability,  survivability,  safety,  and  affordability.” 
DoD  Directive  (DoDD)  5000.01,  The  Defense  Acquisition  System  (Washington,  DC:  Under  Secretary  of 
Defense  (Acquisition,  Technology  and  Logistics),  May  12,  2003).  Certified  current  as  of  November  20, 
2007. 
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conduct  missions  and  survive.  It  was  developed  to  assist  practitioners  in 
quantifying  weapon  system  maturity;  characterizing  mission  capability;  and 
projecting  operational  effectiveness,  suitability,  and  survivability. 

The  proposed  “A  Analytic  Model”  and  “Mission  Stream  Analysis”  are  cross-domain 
tools  that  could  enhance  the  foundational  analysis  of  a  program;  aid  in  devolving  the 
envisioned  mission  and  system  capability  requirements  into  a  system’s  technical 
performance  and  operator  workload  requirements;  and  help  minimize  the  “delta”  between 
domains  across  the  system’s  lifecycle,  from  program  definition  through  design, 
development,  and  test. 
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2.  Mission  Stream  Analysis 


Mission  stream  analysis  considers  all  mission-related  tasks  a  weapon  system  is 
expected  to  accomplish  during  an  end-to-end  mission,  including  execution  of  all 
offensive  kill-chain  functions  to  engage  an  enemy  threat  and  execution  of  ah  defensive 
capabilities  to  survive  an  enemy  threat.  The  term  “mission  stream  analysis”  was  coined 
by  IDA’s  Cost  Analysis  and  Research  Division,  and  is  not  in  general  use  in  the  DoD 
acquisition  community;  however,  mission-based  testing  and  kill  chain  analysis  is  used  by 
the  Navy  Operational  Test  and  Evaluation  Force  and  other  Service  test  organizations. 
Mission  stream  analysis  is  intended  to  assist  Systems  Engineers  (SEs)  and  Test  and 
Evaluation  (T&E)  practitioners  in  precisely  characterizing  observed  system  performance 
(i.e.,  test  results)  in  terms  of  mission  capabilities  that  relate  to  effectiveness.  In  practice, 
mission  stream  analysis 

•  Provides  a  high-level  depiction  of  mission  sets  and  the  associated  tasks  required 
to  accomplish  operationally  realistic  missions;3 4 

•  Provides  developers  with  a  basis  for  devolving  operational  concepts  and 
missions  into  sets  of  specific  tasks,  perfonnance  attributes,  and  technical 
performance  and  mission  effectiveness  measures;5 

•  Provides  a  methodology  to  explore  a  system’s  mission  tasks  and  objectives  to 
identify  inconsistencies  and  gaps  between  user  requirements  and  the 
performance  parameters  used  to  design,  develop,  and  test  and  evaluate  the 
system;  and 


3  COMOPTEVFOR  Instruction  3980. 2C,  Code  01A  (Norfolk,  VA:  Department  of  the  Navy,  Commander 
Operational  Test  and  Evaluation  Force,  April  14,  2014). 

4 

The  authors  believe  that  the  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  Manual, 
2012,  provides  a  foundation  for  mission  stream  analysis  and  associated  success  criteria;  the  Manual 
states  “The  CBA  must  also  develop  criteria  for  adequate  mission  performance.  Quantitative  criteria  for 
mission  success  must  be  established  to  support  the  assessment  of  the  materiel  reliability  characteristics 
of  potential  materiel  solutions.  In  most  cases,  these  criteria  will  not  be  simple  pass-fail  standards,  but 
instead  will  represent  a  continuum  of  values.” 

5  The  DoD  Architectural  Framework  (DoDAF)  Version  2.02  provides  insight  into  tasks,  activities,  and 
data  exchanges,  but  does  not  provide  the  performance  and  effectiveness  measures;  more  specifically, 
DODAF  states  “DoDAF-described  Models  in  the  Operational  Viewpoint  describe  the  tasks  and 
activities,  operational  elements,  and  resource  flow  exchanges  required  to  conduct  operations.  A  pure 
operational  model  is  materiel  independent. .  .it  may  be  necessary  to  include  some  high-level  system 
architectural  data  to  augment  information  onto  the  operational  models.”  http://dodcio.defense.gov 
/Portals/O/Documents/DODAF/DoD  AF_v2-02_web.pdf. 


3 


•  Provides  an  analytic  framework  that  links  major  development  and  test  phases. 

The  Joint  Staffs  Capabilities-Based  Assessment6  is  a  primary  source  of  information 
to  support  the  development  of  the  proposed  system’s  mission  streams,  since  it 

begins  by  identifying  the  mission  or  military  problem  to  be  assessed,  the 
concepts  to  be  examined,  the  timeframe  in  which  the  problem  is  being 
assessed,  and  the  scope  of  the  assessment.  A  CBA  determines  the  relevant 
concepts,  CONOPS  [Concept  of  Operations],  and  objectives,  and  lists  the 
related  effects  to  be  achieved. 

Figure  1  depicts  a  mission  stream  for  a  fighter  aircraft  as  a  series  of  mission  tasks, 
from  mission  planning  through  assessing  mission  outcome.  In  this  example,  the  fighter’s 
offensive  kill-chain  functions  (find,  fix,  track,  target,  and  engage  the  enemy  threat)  is  a 
central  concept.  The  figure  also  includes  the  threat  fighter’s  defensive  capabilities  to 
deny,  defeat,  and  survive  the  offensive  kill  chain  associated  with  the  engagement.  End-to- 
end  missions  may  involve  execution  of  many  tasks,  multiple  offensive  kill  chains,  and 
deny/defeat/  survive  multiple  threat  offensive  kill  chains. 


US  Fighter 


Plan 


Enroute 


Ingress 


Offensive  Kill  Chain 

I  Track 


Target 


Defeat 


Defensive  Capabilities  -  Survivability  Spectrum 


Deny 


Figure  1.  Mission  Stream  Analysis 


In  general  terms,  mission  effectiveness  requires  successful  execution  of  system 
functions  (e.g.,  completing  critical  mission  tasks  and  all  segments  of  the  offensive  kill 
chain,  such  as  engaging  threat  fighters,  or  surviving  a  threats-offensive  kill  chain). 
Success,  in  turn,  depends  on  the  technical  capabilities  of  sub-systems  and  the  operators’ 
ability  to  perfonn  their  roles  within  the  mission  timelines. 

A  model  of  a  notional  mission  stream  associated  with  a  fighter/missile  offensive 
engagement  against  a  threat  fighter  is  shown  in  Figure  2.  In  this  example,  a  US  fighter  is 
engaging  an  enemy  threat  fighter  with  a  long  range  air-to-air  missile.  The  US  fighter’s 
offensive  kill  chain  starts  with  the  pre-missile  launch  activities,  which  lead  to  the  launch 
of  the  missile  (engage  function),  followed  by  the  missile  offensive  kill  chain  that 
culminates  in  a  kill  of  the  threat  fighter.  In  this  example,  the  objective  of  the  threat  fighter 


The  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  Manual,  2012. 
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is  to  use  its  defensive  capabilities  to  deny  the  US  fighter  engagement,  or,  if  the  missile  is 
launched,  to  defeat  the  missile  and  survive. 


Threat  Fighter 


US  Fighters  Offensive  Kill  Chain  Objective  -  Engage/  launch  missile  at 

threat  fighter  within  timeline  constraints 

•  Find,  Fix,  Track,  Target,  and  Engage  threat  fighter 

•  Counter  threat  fighters  defensive  capabilities  throughout  kill  chain  functions 

•  Pass  threat  characterization  &  targeting  data  to  missile 

•  Engage/launch  missile  &  update  missile  in  flight  (via  datalink) 

Missile  Offensive  Kill  Chain  Objectives  -  Kill  threat  fighter 

■  Find,  Fix,  Track,  Target  as  required 

•  Configure  based  on  threat  data 

•  Guide  to  threat  fighter  via  datalink  until  active  guidance  conditions  achieved 

•  Counter  threat  fighters  defensive  capabilities  throughout  kill  chain  functions 

•  Fuze  on  threat  fighter,  detonate  warhead,  and  kill  threat  fighter 


Threat  Fighter  Defensive  Capabilities  employed  to  deny 'defeat  survive 

US  fighter/missile  offensive  kill  chains 

•  Physical  size 

•  Radar  Cross  Section  (RCS) 

•  IR  and  electro-optical  signature 

•  Aero  performance  (speed  and  maneuverability) 

■  Electronic  Emissions  ( e.g .,  IFF,  Comm,  etc) 

•  Electronic  attack  (includes  Electronic  Countermeasure  (ECM)  techniques 
which  affect  missile  guidance  and/or  fusing) 

•  Countermeasures 


Figure  2.  Notional  Fighter/Missile  Offensive  Engagement 


The  US  fighter’s  air-to-air  mission  stream  starts  with  mission  planning  and  continues 
with  ground  operations,  take-off,  enroute  tasks  (e.g.,  navigation  and  re-fueling),  and 
ingress  into  the  engagement  area. 

A  more  detailed  view  and  objectives  of  the  US  fighter/missile  engagement’s 
offensive  kill  chains  that  compete  with  the  threat  fighter’s  defensive  capabilities  is  shown 
in  Figure  3. 7 


US  Fighter  Offensive  Kill  Chain 

^Find  t  FiXj  Tracl^^ge^^^^e 


Timeline  to  Engage/Launch 


Fix  | 


Missile  Offensive  Kill  Chain 


Timeline  to  Kill 


Deny 

I  - 


Defeat  Survive 


Defensive  Capabilities  - Survivability  Spectrum 


Threat  Fighter 


Figure  3.  Notional  US  Fighter/Missile  Offensive  Kill  Chains  vs.  Threat  Fighter  Defensive 

Capabilities 


Notional  for  illustrative  purposes  to  help  frame  the  concept. 
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•  US  Fighters  Offensive  Kill  Chain  Objective  -  Engage/launch  missile  at  threat 
fighter  within  timeline  constraints 

-  Find,  fix,  track,  target,  and  engage  threat  fighter 

-  Counter  threat  fighter’s  defensive  capabilities  throughout  the  kill  chain 
functions 

-  Pass  threat  characterization  and  targeting  data  to  missile 

-  Engage/launch  missile  and  update  missile  in  flight  (via  datalink) 

•  Missile  Offensive  Kill  Chain  Objective  -  Kill  threat  fighter 

-  Find,  fix,  track  target  as  required 

-  Configure  based  on  threat  fighter  data 

-  Guide  to  threat  fighter  via  datalink  until  active  guidance  conditions  achieved 

-  Counter  threat  fighter’s  defensive  capabilities  throughout  the  kill  chain 
functions 

-  Fuse  on  threat  fighter,  detonate  warhead,  and  kill  threat  fighter 

•  Threat  Fighter’s  Defensive  Capabilities  -  Employ  across  the  survivability 
spectrum,  to  deny/defeat/survive  the  US  fighter/missile  offensive  kill  chains. 
These  defensive  capabilities  include: 

-  Physical  size 

-  Radar  Cross  Section  (RCS) 

-  Infrared  (IR)  and  electro-optical  signature 

-  Aero  perfonnance  (speed  and  maneuverability) 

-  Electronic  emissions  (e.g.,  Identification,  Friend  or  Foe  (IFF)  and 
Communications) 

-  Electronic  attack  (includes  Electronic  Countenneasure  (ECM)  techniques, 
which  affect  missile  guidance  and/or  fuzing) 

-  Countermeasures 

-  Decoys 

In  this  notional  example,  the  US  fighter’s  air-to-air  mission  stream  ends  with  egress  from 
the  engagement  area,  numerous  tasks  on  return  to  base  (RTB)  (e.g.,  navigation  and  re¬ 
fueling),  landing,  and  assessment  and  debrief. 

The  next  step  in  mission  stream  analysis  is  to  align  the  US  fighter/missile 
capabilities  with  the  engagement  sequencing  across  the  kill  chains  (Figure  4).  The 
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engagement  sequence  includes  the  US  fighter  pre-launch-launch  phase  followed  by  a 
three  phase  post-missile  launch  sequence:  mid-course  (semi-active),  mid-course  (active), 
and  end  game.  In  this  case,  the  model  links  the  mid-course  (semi-active)  phases  of  the 
engagement  sequence  to  both  the  fighter  and  missile  capabilities  using  a  datalink  for 
guidance  commands.  Once  the  missile  is  active,  the  lighter  capabilities  no  longer  support 
the  missile’s  mid-course  (active)  and  end-game  phases  of  the  engagement  sequence.  The 
grayed-out  capabilities  are  those  systems  that  do  not  contribute  to  the  particular  segment 
of  the  engagement  sequence.  This  mapping  of  lighter/missile  capabilities  to  kill  chain 
functions  can  be  useful  to  both  SEs  and  T&E  practitioners  in  the  development  of  test 
events  or  to  perform  system  maturity  assessments. 


Data-Link 
Radar 

IR/Electro-optical  sensors 
Aero  performance 
Electronic  Emissions  Detection 
Electronic  attack/  ECM 

Mission  Computer  (fuse,  classify, 
etc.) 

Engage  -  Launch 


Data-Link 

Radar 

ECCM 

Aero  performance 

Fuse 


Missile  Capabilities 


Data-Link 

Radar 

ECCM 

Aero  performance 


Radar 

ECCM 

Aero  performance 
Fuse 


Mid-Course  (semi-active) 


Mid-Course  (active) 


End  Game 


Pre-Launch;  US  Fighter  Offensive  Kill  Chain 
y  Find  |  Fix  |  Track  |  Find 

Timeline  To  Engage/Launch 
Deny 


Post  Launch  -  Missile  Offensive  Kill  Chain 


Timeline  To  Kill 


Defeat 


Threat  Fighter’s  Survivability  Spectrum 


Figure  4.  Notional  US  Fighter/Missile  Capabilities  Mapped  across  the  Engagement 

Sequence 


In  order  for  the  US  fighter/missile  engagement  to  be  successful,  it  must  “fight” 
through  the  threat  fighter’s  defensive  capabilities.  These  capabilities  can  also  be  aligned 
with  the  threat  fighter’s  survivability  spectrum  that  is  intended  to  deny,  defeat,  and 
survive  the  US  fighter’s  offensive  engagement  sequence  (Figure  5).  The  grayed-out 
capabilities  are  those  threat  systems  that  do  not  contribute  to  defeating  a  particular 
segment  of  the  engagement  sequence. 
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lUS  Fighter  Offensive  Capabilities 


Missile  Capabilities 


Engage  -  Launch 


Deny 

I 


Mid-Course  (semi-active) 


Mid-Course  (active) 


Defeat 


Threat  Fighter's  Survivability  Spectrum 


Physical  size 

Physical  size 

Radar  Cross  Section  (RCS) 

Radar  Cross  Section  (RCS) 

Radar  Cross  Section  (RCS) 

Radar  Cross  Section  (RCS) 

IR  and  electro-optical  signature 

IR  and  electro-optical  signature 

IR  and  electro-optical  signature 

IR  and  electro-optical  signature 

Aero  performance 

Aero  performance 

Aero  performance 

Aero  performance 

Electronic  Emissions 

Electronic  Emissions 

Electronic  Emissions 

Electronic  attack/  ECM 

Electronic  attack/ ECM 

Electronic  attack/ ECM 

Countermeasures 

Countermeasures 

Countermeasures 

Countermeasures 

Decoys 

Decoys 

Decoys 

Decoys 

Threat  Fighter  Defensive  Capabilities  Across  the  Survivability  Spectrum 

Figure  5.  Notional  Threat  Fighter’s  Defensive  Capabilities  Mapped  across  the  Engagement 

Sequence 


Mission  stream  analysis  can  then  present  a  composite  of  the  engagement,  which 
includes  an  alignment  of  US  fighter/missile  offensive  capabilities  intended  to  kill  the 
threat  with  the  threat  fighter’s  defensive  capabilities  intended  to  deny/defeat/survive  the 
engagement  (Figure  6).  This  composite  representation  is  useful  when  defining  both  US 
fighter  and  missile  performance  and  effectiveness  criteria  for  the  developer  and  tester. 


US  Fighter  Offensive  Capabilities 


Data-Link 
Radar 

IR/Electro-optical  sensors 
Aero  performance 
Electronic  Emissions  Detection 
Electronic  attack/  ECM 

Mission  Computer  (fuse,  classify, 
etc.) 


Pre-Launch- Launch 


Missile  Capabilities 

Data-Link 

Radar 

Radar 

Radar 

ECCM 

ECCM 

ECCM 

Aero  performance 

Aero  performance 

Aero  performance 

Fuse 

Fuse 

Mid-Course  (semi-active) 

Mid-Course  (active) 

End  Game 

Pre-Launch:  US  Fighter  Offensive  Kill  Chain 

|  Track  |  Tar 


Post  Launch  -  Missile  Offensive  Kill  Chain 
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Timeline  To  Engage/Launch 
Deny 


Timeline  To  Kill 


Threat  Defensive  Capabilities  Across  the  Survivability  Spectrum 


■X 


Physical  size 

Physical  size 

Physical  size 

Radar  Cross  Section  (RCS) 

Radar  Cross  Section  (RCS) 

Radar  Cross  Section  (RCS) 

Radar  Cross  Section  (RCS) 

Aero  performance 

Aero  performance 

Aero  performance 

Aero  performance 

Electronic  attack/ ECM 

Electronic  attack/  ECM 

Electronic  attack/  ECM 

Electronic  attack/  ECM 

Countermeasures 

Countermeasures 
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Figure  6.  Notional  Alignment  of  US  Fighter/Missile  Offensive  Capabilities  vs  Threat 
Fighter’s  Defensive  Capabilities  Mapping 


8 


From  these  relationships,  the  developers  and  testers  can  develop  a  relational 
template  of  the  engagement  sequence,  such  as  the  one  shown  in  Figure  7.  This  template  is 
populated  with  the  threat-defensive  capabilities  that  effect  the  sequencing  and  execution 
of  the  fighter’s/missile’s  offensive  kill  chain;  this  may  be  useful  with  respect  to  test 
design  or  assessments.  For  example,  if  the  developer  and  tester  were  concerned  about  a 
radar  missile’s  active  mid-course  capabilities,  they  should  focus  the  target  configuration 
on  replicating  the  threat  defensive  capabilities  (e.g.,  radar  cross  section,  aero 
performance,  electronic  attack,  countermeasures,  and  decoys). 


US  Fighter/Missile  Offensive 

Fighter  Pre-Launch  Kill  Chain 
vs  Threat  Fighter  Defensive  Capabilities 

Post-Launch  Missile  Kill  Chain 
vs  Threat  Fighter  Defensive  Capabilities 

Engagement  Analysis 

Find 

Fix 
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Target 
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(Launch) 
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Fix 
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Target 

Kill 

Fighter  -  Pre-Launch 

2-8 

2-8 

2-8 

2-8 

2-8 

Post  Launch  (Fighter  -  Data 
Link)  -  Missile  semi-active 

2-8 

2-8 

Missile  -  active  -  Mid  Course 

2,4, 6-8 

2, 4, 6-8 

2, 4,6-8 

2,4, 6-8 

Missile  -  End  Game  Kill 


1,4,7, 8 


Threat  Fighter  Defensive  Capabilities  that  impact  US  Fighter/Missile  offensive  kill  chains  continuity  include: 

1.  Physical  size  (desired  for  end-game  fusing  analysis) 

2.  Radar  Cross  Section  (RCS) 

3.  IR  and  electro-optical  signature 

4.  Aero  performance  (speed  and  maneuverability) 

5.  Electronic  Emissions  (e.g.,  IFF,  Comm,  etc.) 

6.  Electronic  attack  (includes  Electronic  Countermeasure  (ECM)  techniques  which  affect  missile  guidance  and/or  fusing) 

7.  Countermeasures 

8.  Decoys 


Figure  7.  Notional  Mapping  of  Threat-Defensive  Capabilities  to  Fighter/Missile  Offensive 

Kill  Chains 


A  similar  analysis  is  conducted  when  the  US  fighter  and  the  threat  fighter’s  offensive  and 
defensive  roles  are  reversed;  when  the  US  fighter  is  under  attack,  it  must  employ  its 
defensive  capabilities  across  its  survivability  spectrum  to  deny,  defeat,  and  survive  the 
threat  fighter/missile  offensive  kill  chains. 

A  mission  stream  analysis  template  for  reporting  results  of  a  US  fighter/missile 
offensive  missile  engagement,  with  regard  to  the  system’s  (fighter’s)  technical 
performance  and  pilot  workload,  is  shown  in  Figure  8.  Tests  must  be  planned  and 
executed  specifically  to  collect  this  kind  of  data  from  a  spectrum  of  data  sources  (e.g., 
hardware-in-the-loop  facilities,  systems  integration  laboratories,  modeling  and 
simulation,  open  air  test  ranges).  The  legend  at  the  bottom  of  the  figure  provides  the  key 
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for  interpreting  the  results  presented.  The  system  performance  scale  shows  the  objective 
and  threshold  values  associated  with  the  performance  requirement,  the  best  performance 
to  date,  and  the  performance  demonstrated  in  this  test  scenario.  The  pilot  workload  scale 
provides  the  limits  of  unacceptable  workload,  little  spare  capacity,  and  insignificant 
workload;  the  values  indicators  are  the  same  as  the  system  perfonnance  scale.  The  color 
indicators  on  the  system  and  pilot  scales  identify  the  test  venue  where  the  data  were 
obtained.  The  test  venues  include  such  sources  as  test  range,  hardware-in-the-loop 
facility,  and  modeling  and  simulation. 
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Figure  8.  Mission  Stream  Analysis  of  System  Technical  Performance  and  Pilot  Workload 
(Notional  Pre-Launch  Offensive  Kill  Chain  (normalized)) 


Now,  when  reading  the  data,  a  large  difference  between  the  system’s  “scenario 
performances”  and  “performance  to  date”  is  an  indication  of  the  system’s  “excess 
capacity”  or  reserve  capability;  “scenario  performance”  reflects  the  end-to-end 
performance  of  each  element  of  the  system’s  kill  chain  for  a  specific  scenario  (i.e., 
requires  successful  completion  of  the  kill  chain  for  the  engagement  sequence); 
“performance  to  date”  reflects  a  composite  representation  of  the  “best”  performance  of 
each  element  of  the  kill  chain,  independent  of  the  scenario,  seen  to  date  (i.e.,  isolated 
performance  of  a  kill  chain  element  relative  to  its  requirement  (threshold  and  objective). 
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In  most  cases,  scenario  testing  does  not  stress  the  technical  capabilities  of  the  system,  but 
it  does  inform  on  the  robustness  of  the  system,  and  provides  an  analytical  basis  for  cost 
performance  trades.  For  instance,  under  the  kill  chain  fix  function,  the  system 
performance  indicates  a  55  percent  excess  capacity  for  this  scenario  with  acceptable  pilot 
workload.  However,  under  the  targeting  function,  the  system  has  not  yet  achieved  its 
performance  specifications  and  only  20  percent  excess  capacity  was  available  in  the  test 
scenario;  additionally,  the  pilot  workload  was  high  in  the  targeting  function,  indicating 
there  may  be  human  factor  issues  associated  with  the  kill  chain  execution. 

In  conclusion,  the  benefits  of  a  mission  stream  analysis  are  that  it  provides  a 
mission-based  presentation  framework  for  senior  leadership  (e.g.,  Milestone  Decision 
Authority  (MDA),  Program  Executive  Officer  (PEO),  Program  Manager  (PM),  and 
warfighters)  and  a  logical  approach  to  structuring  an  efficient  test  program  that 
emphasizes  the  collection  of  essential  data  to  support  key  acquisition  decisions.  A 
mission  stream  analysis  may  assist  senior  leadership  by  providing  mission  context  for 
cost  perfonnance  trades,  and  for  reporting  test  results  and  system  maturity,  specifically 
with  regard  to: 

•  Assessing  test  progress  and  current  system  performance  against  mission 
requirements 

•  Aligning  program  management  priorities  with  mission  requirements,  particularly 
those  relating  to  mission  effectiveness,  suitability,  and  survivability 

•  Providing  a  foundation  for  mission-based  cost-perfonnance  trades  and  T&E 
planning  and  execution 

Mission  stream  analysis,  as  a  cross-domain  tool,  could  enhance  the  foundational 
analysis  of  a  program;  aid  in  devolving  the  envisioned  mission  and  system  capability 
requirements  into  a  system’s  technical  perfonnance  and  operator  workload  requirements; 
and  help  minimize  the  “delta”  between  domains  across  the  systems  lifecycle,  from 
program  definition  through  design,  development,  and  test  and  evaluation,  as  described  in 
the  next  section  of  the  paper. 
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3.  Role  of  Mission  Stream  Analysis  in  DoD 
Acquisition  -  A  Analytic  Model 


A.  The  Acquisition  Process 

o 

DoD  Directive  5000. OT  states:  “The  primary  objective  of  Defense  acquisition  is  to 
acquire  quality  products  that  satisfy  user  needs  with  measurable  improvements  to  mission 
capability  and  operational  support,  in  a  timely  manner,  and  at  a  fair  and  reasonable 
price.”  In  essence,  DoD  develops  weapon  systems  to  effectively  conduct  missions  and 
survive.  The  process  normally  begins  with  combatant  commanders  identifying  mission 
needs  and  capability  gaps/shortfalls.  The  process  continues  with  the  Joint  Requirements 
Oversight  Council  (JROC)  validating  those  mission  needs;  identifying  a  requirement  for 
a  material  solution;  and  identifying  the  key  performance  parameters  needed  by  a  material 
solution  to  mitigate  the  gap.  The  focus  is  always  on  the  mission.  Once  the  requirement  is 
validated,  the  acquisition  process  is  focused  on  the  development  of  a  weapon  system.  An 
integral  part  of  that  development  process  is  the  verification  and  validation  of  its 
performance  and  the  demonstration  of  its  mission  effectiveness.  Ultimately  it  is  the  T&E 
professionals  who  must  accurately  summarize  the  system’s  and  the  operator’s  capabilities 
and  limitations  so  that  decision  makers  can  reasonably  judge  the  relationship  of 
demonstrated  system  performance  to  the  desired  mission  capability  (effectiveness  and 
suitability).  These  issues  are  not  trivial. 

The  term  “mission  effectiveness”  is  defined  as  a  measure  of  the  overall  ability  of  a 
weapon  system  to  accomplish  a  mission  when  used  by  representative  personnel  in  the 
operational  employment  of  the  system  considering  organization,  doctrine,  supportability, 
survivability,  vulnerability,  and  threat.9  In  the  context  of  mission  stream  analysis,  it  is  the 
ability  of  the  weapon  system  to  successfully  execute  its  offensive  “kill  chains”  and 
survive  the  enemy’s  kill  chain  when  conducting  missions  under  operational  conditions 
and  in  operationally  relevant  environments.  Each  mission  set  should  have  at  least  one 
defined  measure  of  effectiveness  (MOE)  from  which  testable  offensive  and  defensive 
performance  attributes  can  be  derived  for  the  purpose  of  assessing  mission 
accomplishment.  Well-defined  effectiveness  measures  are  essential  elements  of  a  credible 
mission  stream  analysis. 


DoDD  5000.01. 

Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and  Development  System,  January  31, 
2011. 
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Mission  stream  analysis  is  an  acquisition  improvement  tool  that  can  help  define 
requirements,  identify  systems  and  sub-systems  that  do  not  add  value  or  contribute  to  the 
mission,  and  can  help  senior  leadership  better  understand  the  mission  capabilities  of  the 
system  under  development,  particularly  with  respect  to: 

•  Aligning  program  management  priorities  with  achieving  mission  capabilities, 

•  Understanding  the  “so  what”  of  test  progress  by  assisting  in  relating  system 
performance  to  mission  effectiveness  and  suitability,  and 

•  Providing  a  mission-based  tool  for  cost-perfonnance  trades. 

Establishing  better  cross-domain  consensus  with  regard  to  missions,  measures,  and 
environments  used  to  develop,  validate,  and  demonstrate  the  system’s  capabilities  to 
accomplish  its  mission  would  aid  in  improving  the  acquisition  process.  The  focus  should 
be  on  minimizing  the  differences,  or  “A”,  in  defining  missions,  measures,  and 
environments  across  domains.  Acquisition  professionals  (engineers,  testers,  and  resource 
analysts)  have  two  immediate  challenges  early  in  the  acquisition  process:  (1)  establishing 
relevant  MOEs  that  will  provide  insight  into  system  performance  and  operator 
capabilities  relative  to  mission  requirements;  and  (2)  defining  the  appropriate  assessment 
activities  necessary  to  evaluate  the  system’s  mission  capabilities  and  limitations. 
However,  developers  and  testers  must  prioritize  their  activities  against  schedule  and 
resource  constraints.  These  prioritization  efforts  must  be  undertaken  within  the  context  of 
delivering  mission  capability  to  the  warfighter. 

Historically,  developers  tend  to  focus  on  achieving  specific  performance 
specifications  in  isolation  of  human  performance  attributes  and  mission  effectiveness  and 
suitability  objectives.  As  a  result,  programs  can  spend  considerable  resources  to  resolve 
shortfalls  that  have  little  impact  on  mission  effectiveness.  Conversely,  developers  can 
overlook  the  resolution  of  requirement  shortfalls  that  are  essential  to  warfighter  needs. 
One  example  involves  the  F/A-18E/F  Super  Hornet  integration  of  the  Joint  Stand  Off 
Weapon  (JSOW)  C-l  variant  that  added  a  Link- 16  datalink  to  the  weapon.  The 
development  program  proved  to  meet  technical  performance  requirements,  but  excessive 
pilot  workload  associated  with  employing  the  weapon  was  not  discovered  until  late  in  the 
development  flight  test  program,  as  noted  in  CDR  McFarland’s  presentation  at  the 
Society  of  Experimental  Test  Pilots  55th  Annual  Symposium.  Below  is  an  excerpt  from 
the  presentation  abstract:10 

New  capability  is  a  quantum  leap  in  warfighting  solutions,  but  the 
flexibility  comes  at  a  cost.  This  cost  is  Aircrew  Workload.  Although  there 
was  a  Simulation  Design  Advisory  Group  early  on  in  the  program,  it  was 


LT  Scott  Johnson  (USN)  and  CDR  Andrew  McFarland  (USN),  “Development  Test  of  JSOW  C-l  on  the 
F/A-18E/F  Super  Hornet,”  Abstract  (paper  presented  at  the  Society  of  Experimental  Test  Pilots  55th 
Annual  Symposium,  Anaheim,  California,  September  21-24,  2011). 
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unfortunately  focused  more  on  the  functionality  of  NEW  and  less  on 
human  factors  evaluation.  This  paper  presents  how  human  factors 
deficiencies  have  been  identified  throughout  various  stages  of  the  test 
program,  and  the  importance  of  identifying  deficiencies  and  solutions  as 
early  as  possible. 

Mission  stream  analysis,  if  used  during  requirements  development  and  system  design, 
would  have  identified  the  importance  of  human  performance  in  design  earlier.  That  is,  it 
would  have  provided  better  precision  in  defining  missions,  measures,  and  environments 
across  domains  with  respect  to  human  integration. 

The  Integrated  Defense  Acquisition,  Technology,  and  Logistics  Life  Cycle 
Management  System  chart  (Figure  9)  outlines  the  key  activities  in  the  DoD  systems 
acquisition  process  that  must  work  in  concert  to  deliver  the  capabilities  required  by  the 
warfighters:  the  requirements  process,  the  acquisition  process,  and  the  program  and 
budget  development  process.  These  processes  often  create  organizational  confusion  and 
stovepipes.  For  instance,  a  system’s  missions,  measures,  and  environments,  as  envisioned 
by  the  warfighters,  are  validated  by  the  JROC  and  documented  in  the  Capabilities 
Development  Document  (CDD).  However,  they  may  be  interpreted  differently  by  the  test 
community  in  their  Test  and  Evaluation  Master  Plan  (TEMP).  The  requirements  and 
T&E  communities  do  not  share  formal  approval  authority  on  these  documents.  As  an 
example,  Dr.  Gilmore,  Director,  Operational  Test  and  Evaluation  (DOT&E),  alluded  to 
the  organizational  confusion  and  stovepipes  in  his  September  13,  2013  memo,  as 
excerpted  below:11 

•  The  fact  that  the  P-8A  can  be  fully  compliant  with  KPP/KSA  [Key  Perfonnance 
Parameter/Key  System  Attribute]  thresholds  while  having  significant  shortfalls 
in  mission  effectiveness  indicates  that  these  “most  essential”  operational 
requirements  were  focused  too  narrowly.  In  this  case,  they  define  supporting 
system  characteristics  or  attributes  that  are  necessary,  but  not  sufficient,  to 
ensure  mission  effectiveness. 

•  The  lack  of  KPPs/KS As  related  directly  to  mission  effectiveness  will  inevitably 
create  a  disconnect  between  the  detennination  of  operational  effectiveness  in 
test  reports  and  the  KPP/KSA  compliance  assessments  that  typically  drive 
program  reviews  throughout  development. 

•  Disconnects  between  KPP  compliance  assessments  and  operational  testing  that 
is  focused  on  characterizing  system  effectiveness  across  the  operational 
envelope  are  not  unique  to  the  P-8A  program.  Another  example  of  this  is  the 
Class  I  Unmanned  Aerial  Vehicle  (Class  I  UAV),  the  Tactical  Unattended 


“P-8A  Poseidon  Multi-mission  Maritime  Aircraft  (MMA)  Increment  1  Key  Performance  Parameters 
memo,”  Operational  Test  and  Evaluation,  September  4,  2013. 
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Ground  Sensor  (T-UGS)  and  the  Urban  Unattended  Ground  Sensor  (UUGS)  of 
the  Army's  Early  Infantry  Brigade  Combat  Team  (E-IBCT)  program.  Those 
components  of  the  E-IBCT  program  met  all  of  the  defined  KPPs,  but  were  not 
operationally  effective  since  they  provided  little  or  no  real  operational  value  to 
the  using  unit  in  the  intended  operational  environment. 
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Source:  Integrated  Defense  Acquisition,  Technology,  and  Logistics  Life  Cycle  Management  System,  Version  5.4,  Defense  Acquisition  University  (DAU), 
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Figure  9.  Integrated  Defense  AT&L  Life  Cycle  Management  System 


B.  Developing  the  A  Analytic  Model 

The  Defense  Acquisition  Guidebook  depicts  a  canonical  Systems  Engineering  “V” 
Model  (Figure  10),  which  reflects  the  DoD  process  used  to  develop  weapon  systems.  The 
“V”  Model  assumes  cross-domain  collaboration  through  an  “Integrated  Product  Team” 
process.  Both  the  Integrated  Defense  Life  Cycle  Management  System  and  the  “V”  Model 
are  schedule-  or  timeline-centric  when  it  comes  to  cross-domain  collaboration.  The 
various  communities  get  involved  when  it  is  their  “time”  or  when  the  process  requires 
their  involvement.  Despite  the  time  and  effort  invested  in  the  DoD  Systems  Engineering 
Process,  things  still  go  wrong  on  programs,  including  misunderstanding  of  the 
requirements,  inadequately  defined  technical  baselines,  erroneous  programmatic  and 
technical  assumptions,  inadequate  developmental  test  programs,  loss  of  corporate 
knowledge,  legislative  and  program  structure  changes,  etc. 
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*  Defense  Acquisition  Guidebook,  Chapter  4,  Figure  4.1. F2  Systems  Engineering  Processes 

Figure  10.  Systems  Engineering  “V”  Model 


It  appears  to  us,  that  the  “V”  model  lacks  emphasis  on  the  “mission”  during 
requirements  decomposition,  which  corrupts  the  integrity  of  the  analytic  foundation 
throughout  the  life  cycle.  Under  these  conditions,  we  see  differences  arise  in  domain- 
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specific  interpretations  of  the  user’s  requirements.  That  is,  the  operational  context  (e.g., 
mission,  kill  chains,  objectives,  scenarios,  measures)  for  which  a  weapon  system’s 
capability  is  defined  (requirements)  may  be  different  from  the  context  in  which  the 
system  is  designed  (developer),  which  may  also  differ  from  the  context  used  to  validate 
the  system’s  capability  and  limitations  (test  and  evaluation).  In  one  aircraft  program, 
OT&E  documented  in  the  TEMP  a  planned  evaluation  for  convoy  and  helicopter  escort 
collateral  missions,  that  was  not  in  the  Capabilities  Production  Document  (CPD).  For 
another  aircraft  program,  the  KPPs  did  not  address  any  of  the  missions  identified  in  the 
mission  statement.  A  UAV  program  reported  operational  availabilities  of  90  percent  on 
systems  that  were  fielded,  but  developmental  tests  were  reporting  failures  and 
maintenance  times  that  could  not  account  for  the  90  percent  rating. 

IDA  proposes  a  A  Analytic  Model  (Figure  1 1),  which  complements  the  “V”  model. 
At  the  base  of  the  A  Analytic  Model  is  the  Analytic  Foundation,  which  includes  key 
elements  of  information  needed  by  the  communities  of  interest  to  execute  their 
responsibilities.  The  tenn  “delta”  was  selected  to  focus  on  eliminating  the  differences  or 
“deltas”  between  communities,  and  between  those  common  elements  and  assumptions 
relating  to  the  system  and  its  missions.  In  order  to  keep  the  differences  like  those  cited 
above  from  developing,  the  Analytic  Foundation  must  be  transparent  and  readily 
available  to  all  communities,  and  all  communities  must  recognize  the  Analytic 
Foundation  as  the  authoritative  source  of  information.  We  envision  a  model  that  is  not 
unlike  Wikipedia  with  permissions. 
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The  A  Analytic  Foundation  elements  include: 

•  Foundational  Analysis  (e.g.,  Capabilities-Based  Assessment  (CBA)  and  Gap 
Analysis) 

•  Leadership  Guidance  and  Decisions 

•  Framing  Assumptions 

•  Missions  (definitions,  objectives,  kill  chains,  CONOPS,  etc.) 

•  Mission  Effectiveness,  Suitability,  and  Survivability  measures 

•  System  Perfonnance  measures 

•  Operator  Performance  measures 

•  Operational  Context  (e.g.,  scenarios,  mission  tasks,  kill  chains,  timelines, 
operating  conditions  and  environment,  and  threats  in  which  the  system  and 
operator  are  to  perform  each  mission  and  each  phase  of  a  mission) 

•  Threats  and  Targets 

•  Others  as  required 

Above  the  Analytic  Foundation  are  all  the  decomposition  and  realization  activities 
as  described  in  the  Systems  Engineering  “V”  Model.  The  A  Analytic  Model  is  the 
“domain-centric”  environment  where  communities  reach  down  to  the  Analytic 
Foundation  for  key  elements  of  information,  regardless  of  the  time  phasing  of  the 
program;  for  instance,  the  operational  context  that  was  used  in  defining  the  requirements 
should  be  available  for  the  developer  and  tester  to  help  ensure  that  the  same  operational 
context  is  used  to  design  and  test  the  system. 

Figure  12  depicts  the  cross-domain  collaboration  and  synergy  enabled  by  the  “A” 
model  throughout  the  life  cycle.  In  this  view  of  the  model,  the  three  communities 
(requirements,  development,  and  test)  are  shown  reaching  into  the  Analytic  Foundation  to 
both  retrieve  and  provide  information. 
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What  it  is  Designed  to  Do 


What  it  Needs  to  Do 


What  it  Docs 


Requirements 


A  Analytic  Foundation  -  Domain  Centric 


Foundation 

Analysis 

Leadership 

Guidance 

Framing 

Assumptions 
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Timelines 
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Operational  Context 
(e.g.,  Scenarios, 
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System  Performance 

Test  and  Evaluation 


Figure  12.  Cross-Domain  Synergy 

Key  A  Analytic  Model  domain  responsibilities  are  described  below: 

•  Requirements  (Real  World):  What  the  System  Needs  to  Do  -  Provides  the 
Foundational  Analysis  and  Assumptions  (e.g.,  CBA  and  Gap  Analysis)  and  real 
world  mission-focused  requirements  development.  Mission  set  should  have  at 
least  one  MOE  defined  from  which  testable  offensive  and  defensive 
performance  attributes  can  be  derived  for  the  purpose  of  assessing  mission 
accomplishment.  System  performance,  suitability,  and  survivability  attributes 
and  operator  capabilities  should  be  defined  with  respect  to  the  projected 
operational  context  (e.g.,  scenarios,  threats,  targets).  The  integrity  of  the 
operational  context  (including  operational  tasks,  events,  durations,  frequency, 
operating  conditions  and  environment,  and  capability  requirements  in  which  the 
system  and  operator  are  to  perform  each  mission  and  each  phase  of  a  mission) 
should  be  maintained.  Operational  concepts  and  missions  should  be  devolved 
into  specific  tasks  (kill  chains),  technical  performance  attributes,  and  mission 
effectiveness  measures: 

-  Capability  gap,  affordability  requirements,  and  need  date 

-  Mission-to-Task  alignment  and  scenario-based  task  performance  standards 

-  Operational  environments,  scenarios,  CONOPS,  performance  attributes,  and 
MOEs 
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•  Development  (Design  World):  What  the  System  is  Designed  to  Do  -  Enables 
system  development  in  a  mission-based  context  based  on  a  better  understanding 
of  the  real  world  and  limitations  of  the  design  world.  This  will  ensure  that  the 
“system”  design  is  fully  integrated  with  the  “operator”  in  accomplishment  of 
realistic  end-to-end  missions,  and  it  aligns  system  and  sub-system  functions  and 
performance  attributes  with  the  specified  tasks  (of  the  kill  chain)  in  the  mission 
context: 

-  Provides  the  technical  framework  and  analytic  tools 

-  Manages  requirements,  to  include  effectiveness,  suitability,  and 
performance  measures 

-  Identifies  and  mitigates  development  risk 

•  Test  and  Evaluation  (Test  World):  What  the  System  Does  -  Validates  the 
system’s  mission  effectiveness,  and  system  perfonnance,  suitability,  and 
survivability  capabilities  and  operator  capabilities  during  T&E  in  the  operational 
context  for  which  the  system  was  envisioned  and  designed.  This  provides  an 
improved  capability  to  quantify  the  differences  between  real  world  and  test 
world  limitations.  It  allows  the  evaluation,  verification,  and  validation  of  system 
capabilities  and  limitations,  ensuring  that  all  required  system  performance  and 
mission  effectiveness  requirements  are  planned  and  tested  in  the  desired 
operational  context  (i.e.,  the  ability  to  successfully  execute  weapon  system  “kill 
chains”  when  conducting  missions  under  operationally  relevant  environments): 

-  Balances  risk  within  the  T&E  scope,  schedule,  and  cost;  develops  the 
system  T&E  strategy  and  test  plans 

-  Associates  KPPs  with  mission  sets  and  the  flowdown  to  test  events 

-  Identifies  key  resource  constraints  (e.g.,  targets,  test  infrastructure) 

-  Highlights  risks  of  proceeding  with  the  test  program  in  cases  in  which  a 
weapon  system  is  not  yet  meeting  a  system  performance  requirement  or 
KPP 

Figure  13  provides  a  template  to  document  the  understandings  of  each  community. 
The  analyst  mines  the  existing  documents  for  the  documented  domain-specific  (rows) 
understanding  of  the  products  (columns)  that  contribute  to  a  mission  effective  rating  and 
enters  the  description  into  the  template.  Once  filled  in,  the  template  provides  evidence  of 
the  existence  or  absence  of  differences. 
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Figure  13.  Minimizing  the  A  across  Domains 

Figure  14  provides  a  specific  example  of  the  use  of  the  template  for  a  fighter 
program’s  “Offensive  Counter  Air”  mission.  We  searched  the  Operational  Requirements 
Document  (ORD)  for  “need,”  the  System  Verification  Plan  (SVP)  for  “design,”  and  the 
TEMP  for  “does.” 

Working  down  Column  2  for  the  Offensive  Counter  Air  (OCA)  mission,  we  found 
the  ORD  did  not  define  operational  context,  in  the  SVP  the  contractor  made  up  his  own 
reference  mission  because  reference  missions  were  not  provided  for  in  the  contract,  and 
the  TEMP  cited  the  Joint  Operational  Test  Team  (JOTT)-defined  missions. 

Searching  for  MOEs  and  Performance  (Column  3),  we  found  that  the  ORD  does  not 
provide  measures  of  effectiveness  for  any  missions,  the  contractor  derived  his  Critical 
Operational  Issues  (COIs)  (and  these  were  not  required  to  track  to  the  COIs  in  the 
TEMP),  and  the  TEMP-defined  MOEs  track  to  JOTT-defined  COIs  based  on  test  team 
rating. 

Searching  for  Operator  Performance  (Column  4),  we  found  that  the  ORD  provided 
for  the  system  to  “Reduce  Pilot  Workload”  without  providing  a  comparative  system,  the 
designer  assessed  workload  in  a  simulator,  and  OT&E  depended  on  pilot  ratings  without 
providing  a  rating  scale  or  thresholds  to  be  met. 

From  this  example,  we  can  see  that  missions,  effectiveness  and  performance  criteria, 
and  operator  workload  expectations  differ  greatly  between  communities. 
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Figure  14.  Notional  Program  that  Highlights  the  A  across  Domains 


The  A  Analytic  Model  is  the  “domain-centric”  environment  where  communities 
reach  down  to  the  Analytic  Foundation  for  key  elements  of  information,  regardless  of  the 
time  phasing  of  the  program;  For  the  A  Analytic  Model  to  work,  the  Analytic 
Foundation  must  be  transparent  and  readily  available  to  all  communities,  and  all 
communities  must  recognize  the  Analytic  Foundation  as  the  authoritative  source  of 
information.  We  envision  a  model  that  is  not  unlike  Wikipedia  with  permissions.  We 
believe  adopting  the  model  will  improve  cross-domain  integration,  communication,  and 
analysis  throughout  the  development  lifecycle  by: 

•  Focusing  on  the  “real  world”  and  maintaining  the  integrity  of  the  analytic 
foundation 

•  Establishing  the  technical  framework,  and  managing  requirements  around 
mission  effectiveness,  suitability,  and  survivability  measures  and  the  associated 
system  perfonnance  attributes 

•  Codifying  the  system’s  Foundational  Analysis  and  Assumptions,  Leadership 
Guidance,  and  Framing  Assumptions  throughout  the  lifecycle 

•  Identifying  and  mitigating  risk 
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4.  Summary 


This  paper  describes  two  cross-domain  analytic  tools  for  enhancing  the  foundational 
analysis  and  the  mission  effectiveness  outcome  on  major  defense  acquisition  programs. 
The  first  tool  is  Mission  Stream  Analysis,  which  aids  in  decomposing  the  mission  and 
system  capabilities/requirements  into  system  technical  performance,  operator  workload 
requirements,  and  metrics  that  contribute  to  demonstrating  mission  effectiveness.  The 
second  tool  is  the  A  Analytic  Model,  which  provides  an  approach  for  identifying 
disparate  interpretations  of  the  systems  requirements  and  metrics  in  the  analytic 
foundation  so  that  the  differences  can  be  eliminated;  i.e.,  the  model  helps  minimize  the 
difference  (delta)  in  interpretations  of  requirements,  performance  parameters,  and  metrics 
by  the  various  communities,  specifically  with  regard  to: 

•  What  the  system  needs  to  do  (requirements  community) 

•  What  it  is  designed  to  do  (development  community) 

•  What  it  does  (T&E  community) 
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